De-duplication deployment planning

ABSTRACT

Assignment of files to a de-duplication domain. Address space of data files is divided into multiple containers. For each of the containers, a file metadata scan is performed to obtain file system metadata, which is aggregated and summarized in a content feature summary. A content feature summary prediction measurement is measured between containers from the generated content feature summary, and files from each container are assigned to a de-duplication domain based upon the content similarity predication measurement.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application is a continuation patent application claiming thebenefit of the filing date of U.S. patent application Ser. No.13/908,955 filed on Jun. 3, 2013 and titled “De-Duplication DeploymentPlanning,” now pending, which is hereby incorporated by reference.

BACKGROUND

The present invention relates to increased efficiency of datade-duplication. More specifically, the invention relates to selectingdata containers for placement into a de-duplication domain.

De-duplication is a method to reduce the number of data storage devicesthat need to be used to store a given amount of information. It operatesby detecting repetition of identical chunks of data, and in someinstances replacing a repeated copy with a reference to another copy ofthe same content. A de-duplication system also provides forreconstructing the original form of content which has been stored in acompressed manner. References are used to locate the original copies ofthe data so that the full-length form of the desired content can bedelivered.

Systems employing de-duplication can experience performance issues whenapplied to large-scale storage systems. To resolve this issue, systemsbuilt for large-scale storage are generally designed to adopt ascale-out strategy such that separate hardware can operate independentlyon separate sub regions of the storage. Operating independently isnecessary so that messaging overheads, lock delays, and blocking waitsdo not grow too large. However, de-duplication imposes a limitation fora dependent operation across its entire span. This limitation creates arequirement that all nodes involved in the full span of de-duplicationbe in frequent messaging contact and block waits that can degradescalability.

BRIEF SUMMARY

This invention comprises a method for assigning data containers to ade-duplication domain.

In one aspect, a method is provided for assigning files from containersto a de-duplication domain. An address space of data is divided intomultiple containers and a file metadata scan of each container isperformed. The file metadata scan obtains file system metadata for allfiles utilizing each of the containers. The file metadata is aggregatedinto characterizations for each metadata dimension, and a contentfeature summary is generated for each container. Content similaritypredication is measured between containers from the generated contentfeature summary. Files are assigned from each container to ade-duplication domain based on the measured content similarityprediction.

Other features and advantages of this invention will become apparentfrom the following detailed description of the presently preferredembodiment of the invention, taken in conjunction with the accompanyingdrawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The drawings referenced herein form a part of the specification.Features shown in the drawings are meant as illustrative of only someembodiments of the invention, and not of all embodiments of theinvention unless otherwise explicitly indicated.

FIG. 1 depicts a flow chart illustrating a method for assigningcontainers to a de-duplication domain.

FIG. 2 depicts a flow chart illustrating a method for computing thediscrete-file portion of a feature summary.

FIG. 3 depicts a flow chart illustrating a method for computing adiscrete-file similarity between containers.

FIG. 4 depicts a flow chart illustrating a method for computing an ownergroup similarity between containers.

FIG. 5 depicts a flow chart illustrating a method for computing a typeand size similarity between containers.

FIG. 6 depicts a block diagram depicting a system for assignment ofcontainers to a de-duplication domain.

FIG. 7 depicts a block diagram showing a system for implementing anembodiment of the present invention.

DETAILED DESCRIPTION

It will be readily understood that the components of the presentinvention, as generally described and illustrated in the Figures herein,may be arranged and designed in a wide variety of differentconfigurations. Thus, the following detailed description of theembodiments of the apparatus, system, and method of the presentinvention, as presented in the Figures, is not intended to limit thescope of the invention, as claimed, but is merely representative ofselected embodiments of the invention.

Reference throughout this specification to “a select embodiment,” “oneembodiment,” or “an embodiment” means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment of the present invention. Thus,appearances of the phrases “a select embodiment,” “in one embodiment,”or “in an embodiment” in various places throughout this specificationare not necessarily referring to the same embodiment.

The illustrated embodiments of the invention will be best understood byreference to the drawings, wherein like parts are designated by likenumerals throughout. The following description is intended only by wayof example, and simply illustrates certain selected embodiments ofdevices, systems, and processes that are consistent with the inventionas claimed herein.

Within a data space, duplicate content does not exist as a homogeneousdistribution. Rather, duplicate content is often irregularly distributedwithin the data space such that pockets of the data space may contain agreater proportion of duplicate content. For purposes of efficientde-duplication, it is advantageous to identify such areas of greaterduplicate content and to reorganize these areas within a de-duplicationdomain. A file is a named entity for organizing data, which may containdata and have additional attributes beyond the data it contains at agiven time. The name of a file serves to identify it and distinguish itfrom other files. In one embodiment, the name might provide to the useran indication of the content or usage of the file. A file may by way ofnon-limiting examples be a document file stored in a computer filesystem, a folder or subdirectory to contain data files, a web page, anobject resident in an object store, a database table, or a row or columnin a database table. Additional attributes may include, but are notlimited to, a name; one or more types; timestamps of creation,modification, and other operations; access controls that determines whocan perform what operations on the contents and which may includeownership information; content tags; access and modification history;performance and usage characteristics; and characteristics associatedwith current or past contents such as the size of contents, frequenciesof different data symbols, language, and application type. The variousattributes of a file may be referred to as its metadata and systems forstoring files generally include provisions for retrieving the metadata.

FIG. 1 is a flow chart (100) illustrating a method for assigning filesto de-duplication domains based on their similarity to one another. Filecollections with similar attributes may be more likely to have datachunks of identical content and files with no similarity arecomparatively unlikely to have many identical data chunks. The addressspace of possible names for files is divided into sections hereafterknown as containers (102). Containers can be individually placed intodifferent de-duplication domains. Placing a container into ade-duplication domain means that the files currently in the containerare placed into that domain, and while the container remains in thede-duplication domain the files in it will continue to reside in thatdomain if their content is changed and future files created in thatcontainer will be placed in that domain. Each file among the data isconfined to a single container and the files in a container arerestricted to a limited number of domains.

A container does not have a fixed limit on the number or total size offiles it may contain. However, there may be limits on what ade-duplication domain can contain, and in one embodiment, the containersmay be constructed so that the content of each is small enough that morethan one container will fit into each domain. In one embodiment thetypical size of content in a container is between one-tenth andone-one-thousandth of the typical size of a domain. In one embodiment,the containers may be configured so that each container has asubstantial population of files in it when the procedure of assigningfile to de-duplication domains is performed, as illustrated in detail inFIG. 1 is performed, so that the estimates of content similarity will bestatistically meaningful. In one embodiment, the number of containers isnot be too large, so that the computational cost of computing similarityvalues over pairs is not too large. For example, in one embodiment thetotal volume of data is between 10 terabytes and 10,000 terabytes andthe number of containers is between 100 and 10,000. The number ofdomains to which the files within a container are confined varies amongembodiments. In one embodiment, individual files or subdirectories areassigned to de-duplication domains without restriction, and the groupinginto containers can be done in a systematic way based on the fileattributes so as to maximize the effectiveness of de-duplication. Theeffectiveness can in general be improved if files likely to have contentsimilarity are in the same container. For example, defining containersby file type, date of creation, owner, or a combination of theseattributes may succeed in placing files with content similarity into thesame container. In one embodiment, system constraints delineateindivisible groups of files that must be restricted to a singlede-duplication domain, and the containers are constructed such that anyindivisible group is placed entirely into one container. Accordingly,files are confined among the containers for placement into ade-duplication domain.

The further steps in the procedure of FIG. 1 provide for determiningmeasurements of similarity between containers based on the attributes ofthe files they contain. File metadata for each container is scanned tocollect the attributes of the files in them (104). One or morecategories of attribute are selected to be dimensions used to generatecomponents of the content feature summary. The metadata for eachdimension is aggregated into a characterization (106). Thecharacterization for a dimension is a data object that represents thedistribution of attribute values from the dimension in a form thatpermits comparison between containers. In one embodiment thecharacterization for a select dimension is a histogram of how many filesor how many bytes of file content are associated with each of a set ofcategories for attribute values in the dimension. For example thecharacterization for the access and ownership dimension may be ahistogram of the distribution over owner groups. In one embodiment thecharacterization for a select dimension is a sampling of attributevalues in the dimension. For example the characterization for the filename dimension may be a discrete file summary consisting of a samplingof records each record corresponding to one file.

Once organized, the metadata is utilized to generate a content featuresummary for each container (108). Specifically, the content featuresummary incorporates the characterizations of the container for each ofthe attribute dimensions. In one embodiment, the metadata results may besupplemented with selective inspection of file content e.g. to resolvefile type uncertainties. In one embodiment, the content feature summaryincludes statistical distributions of files and file volumes accordingto criteria associated with content similarity. The statisticaldistributions may include, but are not limited to, one or more of thefollowing characterizations, distribution of file extensions and filetypes, distribution of file name characteristics, distribution of filesizes, distribution of owners and owning groups, and distribution ofcreation times. The distribution for any given dimension is measured andencoded based on one or more of these characterizations. Accordingly,the metadata among containers is utilized to determine content featuresummaries of the containers.

Content similarity measurements are computed from one or more of thecontent feature summaries (110). Content similarity measurements mayinclude, but are not limited to, a discrete-file similarity as depictedand described in FIG. 3, an owner group distribution similarity asdepicted and described in FIG. 4, and a type and size distributionsimilarity as depicted and described in FIG. 5. In one embodiment, eachstatistical distribution is compared to a like statistical distributionof the same type, and the degree of similarity between the distributionsis assessed. A degree of similarity across multiple distributionsdetermines an overall predicted content similarity between containers.Accordingly, the degree of similarity between containers is assessedthrough content similarity measurements determined from one or morevarious similarity distributions.

The containers are grouped, or otherwise organized, according tostrength of the content similarity (112). In one embodiment, containersthat meet a designated threshold of content similarity are groupedtogether. In one embodiment, the grouping of the containers isrestricted by constraints which may include but are not limited to, thenumber of files, the combined size of the files, the numbers ofcontainers, etc. These constraints may be implemented to maximize theaggregate predicted content similarity across containers in eachcontainer group, and/or combined over all groups. In one embodiment, thediscrete file similarity values are given a higher priority for groupingthan other similarity values. In one embodiment, maximizing theaggregate of discrete file similarity values fully determines thegrouping of containers. In such an embodiment, other similarity metricsneed not be calculated and/or implemented. In one embodiment,exclusively applying the discrete file similarity values achievesmultiple possible groupings and a single grouping solution is achievedthrough application of the other similarity metrics. In one embodiment,owner group distribution similarity analysis is applied to resolveambiguity of multiple grouping solutions and where multiple groupingsolutions exist subsequent to the distribution similarity analysis, thetype and size distribution similarity analysis is applied. Accordingly,containers are compared and a content similarly measurement is utilizedto group like containers.

It is recognized that statistical similarity is an imperfect indicator.It is understood that there could be a strong statistical similaritywhere there is no shared content. However, where a significant quantityof shared content exists, there is a high likelihood of statisticalsimilarity. If the planning method succeeds in placing all containerswith strong statistical similarity as a group into the same domain,there is a high likelihood that files with much shared content will betogether. If some container groups with strong statistical similarityare split across different domains, some de-duplication benefit might belost. The expected amount of de-duplication benefit loss rises with thenumber of similar containers that need to be split. By reducing thenumber of such container groups that are split across different domains,the statistical-similarity part of the placement method increases theexpectation value of de-duplication benefits.

The containers are assigned into de-duplication domains responsive totheir grouping (114). In one embodiment, the assignment of files withincontainers includes grouping containers with high similarity into acommon de-duplication domain. In one embodiment, the assignment ofcontainers into a de-duplication domain is determined exclusively by thesimilarity considerations described above, in addition to the totalcapacity limit of the de-duplication domain. In another embodiment,additional criteria are applied to the grouped containers for placementconsideration. For example, each domain may have limits along multipledimensions, such as on the total number of operations per second, thetotal megabytes per second, the total number of containers, and thetotal volume of data. Various methods for finding acceptable solutionsto such multi-dimensional bin packing limitations are known in the art.In one embodiment, the containers are grouped into clusters of anintermediate size between the container size and the maximum size ofdomains, according to the de-duplication similarity. In this embodiment,the containers with the greatest similarity are placed into likeclusters, and the clusters are placed into domains using bin-packingmethods. Accordingly, the grouped containers are assigned ade-duplication domain corresponding to their grouping.

As shown in step (108), a content feature summary is generated for eachcontainer in order to aid in the determination of a content similaritymetric. One generated component of the content feature summary is adiscrete file summary. The discrete file summary includes a list ofrecords, with each record corresponding to a file. FIG. 2 is a flowchart (200) illustrating a method for computing the discrete-fileportion of a feature summary for one container. A selection window isinitialized and a retained signature list is initialized as an emptylist (202). The selection window is a data element or data structurethat defines which of the signature values generated in the processdescribed below are to be held in the retained signature list. Signaturevalues which are designated to be held may be described as being in theselection window. Metadata for a next file within a container isselected (204). Features are extracted from a name of the selected file(206) and also from the size of the selected file (208). In oneembodiment, the feature extraction on the name might use a restrictedlength or apply pattern matching to ignore character strings insertedfor distinguishing versions of a file. Accordingly, a file within acontainer is selected and features are extracted therefrom.

At least one signature is computed from the extracted features (210). Asignature is a representation of the features as a data object, with aproperty that two files with all features identical will also haveidentical signatures. A signature also has the property that if twofiles have identical signatures then in most cases their features willbe either identical or substantially similar. In one embodiment, thesignature contains a numerical value in a fixed range, e.g. 0 to 1,computed from extracted features using a hash function. Hash functionsthat can be used for this purpose are known to those of ordinary skillin the art and include functions named MD5, SHA-1, and SHA-256. In oneembodiment, the signature consists entirely of the contained numericalvalue. In one embodiment, the selection window is represented as anumerical threshold in the range of possible signature values, with themeaning that signatures whose contained numerical value is numericallyless than the threshold are to be held in the retained signature list.If the signature is in the selection window, it is added to the retainedsignature list (212). In one embodiment, multiple signatures arecomputed from the extraction of multiple features from a single file andan average factor by which files are over represented is determined andrecorded with the feature summary. The number of signatures generatedfor a file could be increased in proportion to the size of the file e.g.generating one distinct signature for each megabyte of data in the file.In one embodiment, distinct signatures are computed by including anindex for the megabyte offset as a feature in the hash calculation. Inone embodiment, where multiple features are extracted from a singlefile, a signature is computed for each extracted feature. In oneembodiment, the signatures are computed to the file name and file size.Accordingly, one or more signatures from the extracted feature(s) arecomputed.

A subset of files is selected for recording in the summary. In oneembodiment, the number of files determined for selection is a fixednumber, a fixed percentage of files, or some combination thereof. It istherefore determined if the list size is greater than a specifiedthreshold (214). In one embodiment, the threshold is the quantity offiles selected for summary recording. A positive response is followed byadjusting the selection window (216). The adjustment generally has theeffect of reducing the number of potential signature values contained inthe selection window. The adjustment is performed so that any potentialsignature value which would have been discarded prior to the adjustmentwill continue to be subject to discard using the selection windowfollowing the adjustment. In one embodiment, the adjustment is performedso as to ensure that the selection windows used for different containershave the property of nesting similarity. Nesting similarity means thatfor any first selection window used for a first container and any secondselection window used for a second container, it is true either that allsignature values in the second selection window are also in the firstselection window, or that all signature values in the first selectionwindow are also in the second selection window. In other words, oneselection window is mathematically a subset of the other. In oneembodiment, adjustment includes decreasing the value of the numericalthreshold, which is one adjustment procedure that ensures nestingsimilarity. The signatures falling inside the selection window areselected and any signatures falling outside the selection window arediscarded from the retained signature list (218).

Following step (218) or a positive response to step (214), it isdetermined if any additional files remain un-scanned (220). A positiveresponse is followed by a return to step (204), and a negative responseis followed by constructing a feature summary from the retainedsignature list and selection window (222). The feature summarysummarizes the contents of the retained signature list. Accordingly, thefiles to be recorded in the summary are selected and a feature summarycontaining the selected files is generated.

Having computed the discrete file portion of the feature summary foreach container, a similarity metric between containers can becalculated. FIG. 3 is a flow chart (300) illustrating a method forcomputing the similarity metric of two containers from the respectivediscrete file similarities. A selection window and retained signaturelist are initialized (302). A merged selection window is determined asthe intersection of selection windows for both containers (304).Elements from each retained signature list of each of the containerscontained in the merged selection window are counted (306). Any elementnot in the merged selection window does not count toward the totals, sothe result is as if each container had been evaluated using the sameselection window. In this step the property of nesting similarity isdesirable because it minimizes the number of elements that must be thusomitted and therefore yields a better sampling of the containercontents. Common elements shared between both of the retained signaturelists are also counted (308). The discrete-file similarity between thecontainers is computed from the element counts and the selection windowsize (310). In one embodiment, the calculated similarity between thecontainers is assessed by assigning to each common element individuallya score between 0 and 1, where 0 signifies no similarity and 1 signifiesa maximum similarity. Accordingly, the common elements between containersignature lists are assessed for computation of a similarity metricbetween the containers.

In one embodiment the similarity score is the total of scores for commonelements, divided by the total number of elements from both retainedsignature lists contained in the merged selection window. Thatsimilarity score is a number between 0 and 1 where 1 indicates maximumsimilarity. Other similarity measurements may be calculated forcontainer grouping. For example, an owner group distribution similaritymay be calculated through comparison of owner group distributions ofcontainers. An owner group is a grouping of parties identified for theexchange of sharable content. An example of an owner group may includebut is not limited to, a group working on a closely related project, agroup sharing media files, a set of computers used for related tasks, aset of computers used when performing related tasks, identified rolesthat map to related tasks, or network addresses used for supplying oraccessing similar content. The parties in the group correspond to anidentifier in the file metadata and/or in the attributes of a container.

The set of owner groups is determined through construction of an ownergroup mapping function which receives as input the metadata for a fileand/or attributes of the container and outputs an identifier value foran owner group. In one embodiment, an appropriate owner group mappingfunction is dependent upon owner identification and access controlinformation used in the file system. In another embodiment, the positionin a directory tree is of most importance. In yet another embodiment,the owner group mapping function is a function exclusively of thecontainer, for instance in a backup system where each backup client is acontainer and the only ownership or access information for a file is itsassociation with a source backup client. Accordingly, a set of ownergroups is one or more groups of identified parties determined throughconstruction of an owner group mapping function.

FIG. 4 is a flow chart (400) illustrating a method for computing thesimilarity between containers through comparing corresponding ownergroup distributions. The owner group distribution for a containermeasures how much content of a container is associated with each memberof a listed set of owner groups. The set of owner groups is determined(402). In one embodiment, an owner group corresponds to a departmentwithin an enterprise identified through financial accounting codes usedto manage chargeback for the costs of storing files. The financialaccounting codes are available through an administrative tool thatprovides an association with specific names and/or addresses throughwhich files are accessed. In one embodiment, the set of owner groups arethe set of accounting codes. In one embodiment, the set of allaccounting codes in use are gathered using automatic or manualclustering methods. In one embodiment, association of each name and/oraddress point to an associated owner group is tabulated in an ownergroup mapping function. In one embodiment, the files in the containersare scanned and associated with owner groups through the own owner groupmapping function. For example, the name and/or address points areattached to accounting codes and subsequently to owner groups,connecting sub-trees within a file system directory structure. In thisinstance, all files in the sub-tree are assigned to the associated ownergroup. Accordingly, a set of owner groups is established and determinedfor comparison of owner group distributions.

Metadata for files in a container are processed (404). Thereafter, eachof the files is associated with an owner group (406). For each ownergroup, the total number of files in a container associated with theowner group is tallied and the owner group distribution is calculated(408). The similarity between pairs of owner group distributions isassessed as an indicator of similarity between associated containers(410). In one embodiment, the owner group distribution for a containeris a collection of values, one for each possible owner group, each valuerepresenting a total number of bytes from files in that containerassociated with that owner group. In one embodiment, the owner groupdistribution similarity between containers is the sum over all ownergroups of overlap bytes within an owner group. Specifically, the overlapbytes for a given owner group is the minimum total bytes between twocompared containers associated with the owner group. As an example, asystem might have owner groups labeled X, Y and Z, and a container Amight contain 500 megabytes (MB) associated with X, 1000 MB associatedwith Y, and 1500 MB associated with Z, and a second container B mightcontain 2500 megabytes (MB) associated with X, 1200 MB associated withY, and 1000 MB associated with Z. The overlap bytes between A and B forowner group X would be 500 MB, for Y would be 1000 MB, and for Z wouldbe 1000 MB, and the sum of overlap bytes would be 2500 MB. In anotherembodiment, the owner group distribution similarity is the sum ofoverlap bytes divided by the maximum of the byte content between the twocontainers. In the foregoing example, the byte content of container A is3000 MB and the byte content of container B is 3700 MB, so in thatembodiment the similarity would be 2500 MB/max(3000 MB, 3700 MB) whichis 2500/3700 or the value 0.676. Accordingly, containers are comparedthrough comparison of calculated owner group distributions associatedwith the containers.

An additional similarity measurement that may be calculated is asimilarity measurement between type and size distributions ofcontainers. This similarity measurement is calculated through comparisonof the type and file size distribution of two containers and computing acorrelation between them. Specifically, the type and size distributionfor a container measures how much of the content of the container isassociated with each member of a listed set of type and size-rangepairs. FIG. 5 is a flow chart (500) illustrating a method for computingthe similarity between containers through comparison of correspondingtype and size distributions. Initially, file types and size ranges formeasuring the distribution are determined (502). In one embodiment, atype of interest is determined by collecting a list of frequently usedfile extensions or otherwise indicated file types in a collection offiles. In one embodiment, rules are manually inputted to group certaintypes into logical types based on file name pattern matching. Forexample, inputted rules may group “html”, “HTML,” “htm”, and “HTM” asone logical type. For each type, a set of size ranges is defined. In oneembodiment, some or all the file sizes are ignored, a method equivalentto implementing a single size range from zero to the maximum supportedsize. In another embodiment, size bins are used for a given type. Thetype and size-range pairs are defined by the list of files and the sizebins. Accordingly, a set of type and size range pairs are determined forcomputing type and size distributions of containers.

The metadata for files in a container are processed (504), and each fileis associated with a type and size-range pair (506). For each type andsize-range pair, the total number of files in a container associatedwith the type and size-range pair is tallied (508) and the type and sizedistribution is calculated (510). The similarity between pairs of typeand size distributions is assessed as an indicator of similarity betweenassociated containers (512). In one embodiment, the type and sizedistribution for a container is the list of absolute numbers tallied,and the similarity between containers is the sum of overlaps oncorresponding type and size-range pairs. In one embodiment, thedistribution for a container is the percentage per type and size-rangepair obtained by normalizing the absolute numbers, and the similarity isa correlation between the number lists. Accordingly, containers arecompared through comparison of calculated type and size-range pairdistributions associated with the containers.

Additional file attributes may be used to determine additional elementsof the content feature summary and then to determine similarity valuesbetween containers. File attributes useful for this purpose mightinclude labels that denote aspects such as the language represented in afile, the usage type for a file, the distribution of symbols used in afile, and the compressibility of the file's content. File attributesused in this way might be stored within file system metadata, inextended attributes stored by the file system, or in separate tables ordatabases. The feature summary may consist of a distribution of how manyfiles or how much content is associated with each of a number of labelvalues.

The processes shown in FIGS. 1-5 may be embodied as hardware components.FIG. 6 is a block diagram (600) illustrating tools embedded in acomputer system to support assignment of files to a de-duplicationdomain. As shown, the system includes a computer (610) provided with aprocessing unit (612) in communication with memory (614) across a bus(616). Data storage (650) is provided in local communication with thecomputer (610) to store received or generated data. In one embodimentthe data storage (640) may be remote with access to the storage providedacross a network (605).

The computer (610) includes a functional unit (640) having one or moretools to support the functionality of the containers with respect todata de-duplication. The tools embedded in the functional unit (640)include, but are not limited to a content manager (644) and anassignment manager (646). In the embodiment illustrated, the data beinganalyzed already is resident in storage which is capable of performingde-duplication. The address space (652) of the storage (650) issubdivided into containers (654). In this example, there are sevencontainers (654 a), (654 b), (654 c), (654 d), (654 e), (654 f), and(654 g), although the quantity shown herein should not be consideredlimiting. Each container is capable of being associated with onede-duplication domain. The storage (650) is shown with twode-duplication domains (660) and (670), although the quantity shownherein should not be considered limiting. As shown, each container hasalready been associated with a de-duplication domain, hereinafterreferred to as domain, either (660) or (670). When a container isassociated with a domain, the file content for its files is stored bythe operation of that domain, with reference to the chunks held in thechunk storage for that domain. In one embodiment, there might becontainers in data storage (650) that are not associated with ade-duplication domain, and the files for such a container are storedelsewhere in the storage.

When a file is to be accessed for reading or writing, the operation isdirected for service to the de-duplication domain to which the file isassigned if there is one, and elsewhere in the storage otherwise. Insome embodiments, the assigned de-duplication domain is identified byfinding the file's location in the address space, identifying theassociated container, and identifying the domain associated with thecontainer. In some embodiments the possible operations include creationof a new file in the associated de-duplication domain.

Associated with the files contained in any container, whether associatedwith a de-duplication domain or not, is the metadata for those files anda content summary. As shown, container (654 a) is associated withmetadata (656 a) and content summary (658 a), container (654 b) isassociated with metadata (656 b) and content summary (658 b), container(654 c) is associated with metadata (656 c) and content summary (658 c),container (654 d) is associated with metadata (656 d) and contentsummary (658 d), container (654 e) is associated with metadata (656 e)and content summary (658 e), container (654 f) is associated withmetadata (656 f) and content summary (658 f), and container (654 g) isassociated with metadata (656 g) and content summary (658 g). Themetadata is generally maintained in memory or on persistent storage,subject to continual update as the file content is changed. The contentsummary is generated by operation of the content manager (644) withreference to the metadata according to the procedures described in FIGS.1-5. In some embodiments, the content summaries are newly generatedwhenever the assignment manager (646) is called upon to make assignmentdecisions, and they are discarded after that use. In other embodiments,the content summaries are retained after they have been used to make anassignment decision, to be used as input to the content manager (644)when it is called upon to make new content summaries after the contentof files in containers has changed. It is possible for a container toexist and to contain files for an extended time before any contentsummary is generated for it.

FIG. 6 depicts an embodiment in which containers are long-lived entitiesthat reside in the storage 650. In other embodiments the containers arecreated whenever the assignment manager (646) is called upon to makeassignment decisions, and are discarded after the decisions are made andthe assignments of files to domains has been performed. In some suchembodiments the assignment of files includes provisions for newlycreated files to be assigned to domains, for example by inheriting theassignment applicable to a parent directory.

Each de-duplication domain (660) and (670) comprises a chunk storage(662), (672); a map (664), (674) that provides, for each file in thedomain, the locations in the chunk storage of the chunks that make upthe file; and a dictionary (666), (676) that provides the locations ofchunks indexed by their content fingerprints. A domain may also compriseits own processing unit, memory, and physical data storage devices,although in some embodiments it might also share processing units,memory, and physical data storage devices with other domains.

As described below, the content manager (644) and the assignment manager(646) perform actions as described with reference to FIGS. 1-5 toassociate containers with specific de-duplication domains, and as aresult to assign the files from a container to the domain associatedwith the container. This may be done in different situations. Onesituation is the deployment of data into a de-duplicating system. Inthis situation, the containers are defined and none of them has anassociation with a domain, so the files are not assigned to domains. Bythe operation of the procedures of the invention, some or all of thecontainers are assigned to domains, and their files likewise areassigned, so that the benefits of de-duplication may be realized.

A second situation is the re-assignment of containers. In thissituation, some or all containers are initially associated with domains.Then by the operation of the invention, new associations are selectedfor some containers, associating some containers with domains differentfrom the initial association, and the new association leads to anincrease in the de-duplication benefits. A re-assignment may bewarranted if the file content in containers has changed, or if thenumber or capacity of domains has changed. A third situation is are-definition of containers. In this situation the boundaries betweencontainers are re-drawn, for example because some have so much contentthat they would better be managed as separate parts, and because othershave so little content that they should be coalesced into a smallernumber of containers. When a new set of containers is formulated, it mayhappen that some new containers have files all assigned to the samedomain so that it is valid to consider the container as associated withthat domain, and it may happen that other new containers have filesassigned across multiple domains so that the container has noassociation to one domain. It is possible to obtain the metadata of eachcontainer, of either type, construct the content summaries, and selectnew associations of containers to domains according to the procedures ofthe invention.

In one embodiment, each container is associated with a commonality ofusage of data. For example, in one embodiment, different users may havedifferent containers. The tools in the form of the content manager (644)and the assignment manager (646) function to deploy de-duplication ofdata. Specifically, the content manager (644) performs a scan of filemetadata of each container, and from the scan generates the contentfeature summary for each container, as shown and described above. Basedon the content feature summary, the content manager (644) measurescontent similarity, an in one embodiment, a similarity predictionmeasurement, between the containers. The assignment manager (646)functions to assign files from each container to one of thede-duplication domains (660) and (680) based on the computed contentsimilarity.

The content manager (644) employs the processing unit (612) to supportassessment of data in the form of the measurements, so that assignmentof data to a specific de-duplication domains mitigates retention ofduplicate data. As described above in FIGS. 1-4, computations areperformed to support the de-duplication. The content manager (644)employs the processing unit (612) to support the process and analysisassociated with the data de-duplication. More specifically, as describedabove in FIGS. 2 and 3, file similarities are assessed among thecontainers. The content manager (644) employs the processing unit (612)to estimate a discrete file similarity through one or more filesummaries, with each file summary having a list of records correspondingto a file, and each record having a signature computed from a file nameand file size. Furthermore, as described above in FIG. 4, a set of ownergroups and similarity among the owners groups are assessed. The contentmanager (644) employs the processing unit (612) to compute an ownergroup distribution similarity and to compute a correlation between thegroups. The computation of the owner group distribution similarityincludes comparison of owner group distribution of containers andassessment of a correlation between the groups. Similarly, theidentification of one or more sets of owner groups includes the contentmanager (644) to process metadata for files in the container, associatedeach file with at least one owner group, and to compute the owner groupdistribution.

Another functionality supported by the content manager (644) isdescribed in FIG. 5 in detail, and pertains to the assessment ofsimilarity between pairs of type and size distribution of files in thecontainers. Specifically, the content manager (644) as supported by theprocessing unit (612), computes type and size distributions of fileswithin containers, compares the distributions between the containers,and computes a similarity between containers based on the comparison.The type and size distribution of each container is a measurement of howmuch content within the container is associated with each member of alisted set of type and size-range pairs. All of the computationssupported by the content manager (644) function to enable the assignmentmanager (646) to place containers with high similarity into a commonde-duplication domain. Accordingly, data among the de-duplicationdomains and their respective containers are placed based on theirassessed similarities.

As identified above, the content manager (644) and the assignmentmanager (646) are shown residing in the functional unit (640) of thecomputer (610). Although in one embodiment, the functional unit (640)and the managers (644) and (646), respectively, may reside as hardwaretools external to memory (614). In another embodiment, the managers(644) and (646), respectively, may be implemented as a combination ofhardware and software in a shared pool of resources. Similarly, in oneembodiment, the managers (644)-(646) may be combined into a singlefunctional item that incorporates the functionality of the separateitems. As shown herein, each of the managers (644)-(646) are shown localto one computer system (610). However, in one embodiment they may becollectively or individually distributed across a shared pool ofconfigurable computer resources and function as a unit to support datade-duplication. The managers (644) and (646) are shown local to thecomputer (610) in communication with storage (650). In one embodiment,the managers may be incorporated into the data storage (650). Asindicated above, storage (650) may be configured as a separatesemi-autonomous unit for performing data storage and incorporating aprocessing unit and memory, and may be configured with the managers(644) and (646) embedded or in communication with the incorporatedprocessing unit and memory. Similarly, in one embodiment, each of thede-duplication domains (660) and (670) may be configured with a separateprocessing unit and/or memory. Accordingly, the managers may beimplemented as software tools, hardware tools, or a combination ofsoftware and hardware tools.

Furthermore, the described features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments. Examplesof the managers have been provided to lend a thorough understanding ofembodiments of the invention. One skilled in the relevant art willrecognize, however, that the invention can be practiced without one ormore of the specific details, or with other methods, components,materials, etc. In other instances, well-known structures, materials, oroperations are not shown or described in detail to avoid obscuringaspects of the invention.

The functional unit(s) described above in FIG. 6 has been labeled withmanagers. The managers may be implemented in programmable hardwaredevices such as field programmable gate arrays, programmable arraylogic, programmable logic devices, or the like. The manager(s) may alsobe implemented in software for processing by various types ofprocessors. An identified manager of executable code may, for instance,comprise one or more physical or logical blocks of computer instructionswhich may, for instance, be organized as an object, procedure, function,or other construct. Nevertheless, the executable of an identifiedmanager need not be physically located together, but may comprisedisparate instructions stored in different locations which, when joinedlogically together, comprise the managers and achieve the stated purposeof the managers.

Indeed, a manager of executable code could be a single instruction, ormany instructions, and may even be distributed over several differentcode segments, among different applications, and across several memorydevices. Similarly, operational data may be identified and illustratedherein within the manager, and may be embodied in any suitable form andorganized within any suitable type of data structure. The operationaldata may be collected as a single data set, or may be distributed overdifferent locations including over different storage devices, and mayexist, at least partially, as electronic signals on a system or network.

Referring now to the block diagram of FIG. 7, additional details are nowdescribed with respect to implementing an embodiment of the presentinvention. The computer system includes one or more processors, such asa processor (702). The processor (702) is connected to a communicationinfrastructure (704) (e.g., a communications bus, cross-over bar, ornetwork).

The computer system can include a display interface (706) that forwardsgraphics, text, and other data from the communication infrastructure(704) (or from a frame buffer not shown) for display on a display unit(708). The computer system also includes a main memory (710), preferablyrandom access memory (RAM), and may also include a secondary memory(712). The secondary memory (712) may include, for example, a hard diskdrive (714) and/or a removable storage drive (716), representing, forexample, a floppy disk drive, a magnetic tape drive, or an optical diskdrive. The removable storage drive (716) reads from and/or writes to aremovable storage unit (718) in a manner well known to those havingordinary skill in the art. Removable storage unit (718) represents, forexample, a floppy disk, a compact disc, a magnetic tape, or an opticaldisk, etc., which is read by and written to by removable storage drive(716). As will be appreciated, the removable storage unit (718) includesa computer readable medium having stored therein computer softwareand/or data.

In alternative embodiments, the secondary memory (712) may include othersimilar means for allowing computer programs or other instructions to beloaded into the computer system. Such means may include, for example, aremovable storage unit (720) and an interface (722). Examples of suchmeans may include a program package and package interface (such as thatfound in video game devices), a removable memory chip (such as an EPROM,or PROM) and associated socket, and other removable storage units (720)and interfaces (722) which allow software and data to be transferredfrom the removable storage unit (720) to the computer system.

The computer system may also include a communications interface (724).Communications interface (724) allows software and data to betransferred between the computer system and external devices. Examplesof communications interface (724) may include a modem, a networkinterface (such as an Ethernet card), a communications port, or a PCMCIAslot and card, etc. Software and data transferred via communicationsinterface (724) is in the form of signals which may be, for example,electronic, electromagnetic, optical, or other signals capable of beingreceived by communications interface (724). These signals are providedto communications interface (724) via a communications path (i.e.,channel) (726). This communications path (726) carries signals and maybe implemented using wire or cable, fiber optics, a phone line, acellular phone link, a radio frequency (RF) link, and/or othercommunication channels.

In this document, the terms “computer program medium,” “computer usablemedium,” and “computer readable medium” are used to generally refer tomedia such as main memory (710) and secondary memory (712), removablestorage drive (716), and a hard disk installed in hard disk drive (714).

Computer programs (also called computer control logic) are stored inmain memory (710) and/or secondary memory (712). Computer programs mayalso be received via a communication interface (724). Such computerprograms, when run, enable the computer system to perform the featuresof the present invention as discussed herein. In particular, thecomputer programs, when run, enable the processor (702) to perform thefeatures of the computer system. Accordingly, such computer programsrepresent controllers of the computer system.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowcharts and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowcharts or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated. Accordingly, the code stream compressionsupports flexibility with respect to decompression, including,decompression of the code stream from an arbitrary position therein,with the decompression being a recursive process to the underlyingliteral of a referenced phrase.

ALTERNATIVE EMBODIMENT

It will be appreciated that, although specific embodiments of theinvention have been described herein for purposes of illustration,various modifications may be made without departing from the spirit andscope of the invention. Accordingly, the scope of protection of thisinvention is limited only by the following claims and their equivalents.

We claim:
 1. A method comprising: dividing an address space of filesinto multiple containers; performing a file metadata scan, includingobtaining attributes for files in each container; aggregating the fileattributes into characterizations for each attribute dimension, andgenerating a content feature summary for each container incorporatingthe characterizations; measuring a content similarity predictionmeasurement between containers from the generated content featuresummary; and assigning files from each container to a de-duplicationdomain based on the computed content similarity prediction measurement.2. The method of claim 1, wherein the file attributes include filesystem metadata.
 3. The method of claim 2, further comprising estimatinga discrete file similarity through use of one or more discrete filesummaries, each discrete file summary including a list of records, eachrecord corresponding to one file, and each record having a signaturecomputed from a file name and a file size.
 4. The method of claim 2,further comprising computing an owner group distribution similarity,including comparing owner group distributions for two or more containersand computing a correlation between the groups.
 5. The method of claim4, wherein the computation of the owner group distribution similarityincludes determining a set of owner groups, processing metadata forfiles in a container, and associating each file with one of the ownergroups, and computing the owner group distribution.
 6. The method ofclaim 2, further comprising computing type and size distributions withincontainers, comparing said distributions between containers, andcomputing a similarity between containers based on the comparison. 7.The method of claim 1, wherein the assignment of files includes placingcontainers with high similarity into a common de-duplication domain.